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I .    INTRODUCTION 

A.  RESEARCH  FOCUS 

This  research  focuses  on  the  problem  solving  behavior  of 
novice  systems  analysts  during  the  systems  design  process.  In 
this  study  the  term  "novice"  is  used  to  describe  someone  who 
possesses  sufficient  knowledge  to  function  in  a  specific 
domain,  but  has  limited  practical  experience  in  that  domain. 
The  findings  will  be  compared  to  the  problem  solving  behaviors 
of  expert  and  novice  financial  analysts  and  expert  systems 
analysts  from  the  literature  (Ramesh, 19  89 ;  Vitalari,  1981). 
The  outcome  of  this  research  identifies  differences  in  problem 
solving  techniques  of  experts  and  novices.  Possible  uses  for 
the  differences  will  be  discussed. 

B.  SYSTEMS  ANALYSTS 

What  does  a  systems  analyst  do?  A  systems  analyst  may  be 
compared  to  a  language  interpreter.  The  systems  analyst 
functions  as  a  mediator  between  the  customer  and  the  computer 
programmers  and  operators  to  facilitate  effective 
communication.  The  systems  analyst  bridges  the  gap  between 
the  language,  needs,  and  culture  of  the  customer  and  those  of 
the  computer  specialists  (both  hardware  and  software) .  Thus 
the  systems  analyst's  is  required  to  be  able  to  interface  with 
the  customer  to  understand  separate  business  functions  and 


their  intertwining  relationships,  and  then  to  be  able  to 
represent  this  information  in  a  format  understandable  to 
computer  specialists  for  developing  the  system.  The  systems 
analyst  is  a  designer,  technician  and  artist  all  in  one. 

1.  Growing  Demand 

In  today's  competitive  environment,  information  has 
become  a  strategic  resource.  Its  prudent  use,  in  many  cases 
is  essential  to  the  very  survival  of  organizations.  This 
information  requirement  has  exponentially  increased  the  demand 
for  systems  capable  of  expeditiously  representing  data  in 
formats  compatible  with  an  organizations'  needs.  As  hardware 
becomes  more  capable  and  users  become  more  aware  of  what 
computers  can  do  for  them,  their  desire  for  newer  and  more 
capable  systems  increases.  This  results  in  the  already 
voluminous  backlog  of  desired  products. 

2.  Difficulties  Meeting  Demand 

The  demand  for  qualified  and  experienced  systems 
analysts  continues  to  grow.  Currently  this  growing  demand  is 
hindered  by  the  decreasing  population  from  which  new  analysts 
can  join  the  work  force,  and  the  extensive  on-the-job 
experience  currently  required  for  a  novice  to  progress  to  the 
level  of  expert  systems  analyst.  The  above  problem  is 
exacerbated  by  the  fact  that  the  size  of  the  population  under 
instruction   required   to   maintain   parity   is   directly 


proportional  to  the  duration  of  the  instruction  (Chi,  Glasser 
and  Farr,  1988)  . 
3 .   Solution 

Many  years  of  training  and  experience  are  required  for 
a  novice  systems  analyst  to  become  an  expert  in  his  field.  If 
properly  addressed,  the  duration  of  this  transition  might  be 
reduced.  One  suggested  solution  could  be  to  identify  the 
problem  solving  differences  exhibited  between  expert  and 
novice  systems  analysts  during  the  design  process.  If  these 
differences  could  then  be  incorporated  into  an  educational  and 
training  process  of  systems  analysts,  the  duration  of  the 
transition  from  novice  to  expert  systems  analyst  could 
possibly  be  decreased.  The  decrease  in  the  duration  of  the 
transition  time  would  increase  the  supply  of  expert  systems 
analysts,  thereby  lessening  the  demand. 

C.   THESIS  ORGANIZATION 

This  thesis  consists  of  five  chapters.  It  is  arranged  to 
allow  the  reader  to  follow  the  development  of  the  research 
from  initial  problem  identification  in  Chapter  I  to  proposed 
solutions  in  Chapter  V. 

Chapter  II  provides  a  detailed  description  of  the  research 
design.  Chapter  III  discusses  the  findings  and  contains  a 
list  of  specific  behaviors  exhibited  by  novice  systems 
analysts  during  this  study.  In  Chapter  IV  the  results  of 
comparative  studies  between  expert  and  novice  systems  analysts 


and  expert/novice  financial  analysts  and  novice  systems 
analysts  are  provided.  Chapter  V,  the  conclusion  suggests 
possible  uses  for  the  expert -novice  differences  identified 
during  the  comparative  study. 


II.   RESEARCH  DESIGN 

A.  RESEARCH  OBJECTIVE 

This  study  views  and  analyzes  the  problem  solving  behavior 
of  novice  systems  analysts  during  a  task  performance.  The 
results  obtained  will  be  compared  with  the  problem  solving 
behaviors  of  expert  systems  analysts  and  expert/novice 
financial  analysts,  as  identified  in  other  studies.  These 
comparisons  will  identify  if  differences  exist  in  the  problem 
solving  behavior  of  novice-novice  or  novice-expert  analysts. 
The  goal  of  this  study  is  to  identify  these  differences. 

The  basis  of  this  research  is  that  the  cognitive  processes 
of  individuals  may  be  identified  by  analysis  of  task 
performance  (Ericsson  and  Simon,  1984)  .  As  the  functionality 
of  the  cognitive  processes  is  similar  to  that  of  an 
information  processing  system,  a  process  model  of  the 
cognitive  processes  may  be  developed  and  analyzed  (Vitalari, 
1981) . 

B.  RESEARCH  METHOD 

The  methodology  used  in  this  study  for  capturing  and 
analyzing  the  problem  solving  behavior  of  novice  systems 
analysts  during  task  performance  is  protocol  analysis.  A 
detailed  description  of  this  technique  is  provided  in  Appendix 
A. 


C.   SUBJECT  SELECTION 

The  first  task  when  selecting  the  subjects  for  the  study 
was  to  identify  a  specific  population  from  which  to  choose  the 
subjects.  It  was  determined  that  the  subject  population 
should  possess  two  specific  attributes.  The  first  was 
exposure  to  the  theoretical  procedures  used  by  systems 
analysts  during  the  systems  design  process.  The  second  was 
minimal  practical  experience  as  systems  analysts. 

The  population  identified  as  possessing  these  two  required 
attributes  was  fifth  quarter  students  in  the  Computer  Systems 
Management  (CSM)  Curriculum  at  the  Naval  Postgraduate  School, 
Monterey,  CA.  The  first  attribute  requiring  exposure  to 
theoretical  procedures  used  by  systems  analysts  during  the 
design  process  was  satisfied  by  the  subjects'  classroom 
exposure  in  the  CSM  curriculum.  While  in  this  curriculum,  the 
entire  population  received  classroom  instructions  on  systems 
design,  database  design  and  management,  decision  support  and 
expert  systems,  programming,  hardware  and  operating  systems 
design,  economic  and  financial  considerations  and  managerial 
issues.  In  addition,  as  part  of  the  instruction,  the  entire 
population  participated  in  group  and  individual  projects  that 
required  practical  application  of  classroom  theory. 

Volunteers  were  solicited.  Eight  students  were  selected 
to  participate  as  subjects.  They  were  all  military  officers. 
The  group  consisted  of  one  Marine  Corps   officer  and  seven 


Naval  officers.  Three  of  the  subjects  were  Navy  Lieutenant 
Commanders  and  the  other  five  were  Navy  Lieutenants  or  their 
Marine  Corps  equivalent. 

Their  undergraduate  degrees  were  diverse  in  nature  ranging 
from  Computer  Science  to  English.  Only  one  subject  had  ever 
taken  more  than  two  computer  related  (specifically 
programming)  courses  prior  to  commencing  their  studies  at  the 
Naval  Postgraduate  School,  Monterey,  CA. 

Prior  to  attending  the  Naval  Postgraduate  School  only  one 
of  the  subjects  had  ever  been  involved  with  a  systems 
development  project  in  any  capacity.  That  exposure  consisted 
of  functioning  as  a  customer  acceptance  testing  representative 
for  the  Navy  on  a  systems  development  project.  The  subjects' 
minimal  experience  satisfied  the  second  essential  attribute. 

Though  not  a  required  attribute,  it  was  interesting  to 
note  that  all  subjects,  when  asked,  responded  that  they  had 
never  to  their  knowledge  participated  in  a  study  that  utilized 
Protocol  Analysis.  Each  indicated  that  this  was  the  first 
experience  where  they  had  been  asked  to  think- aloud  as  they 
performed  a  task. 

D.   TASK  REVIEW 

The  study  was  limited  to  the  task  of  developing  functional 
specifications  during  the  Information  Requirements 
Determination  (IRD)  phase  of  the  systems  development  life 
cycle.    The  IRD  phase  may  be  considered  to  be  the  most 


important  segment  of  any  systems  development  project. 
However,  its  importance  is  often  overlooked  or  misunderstood 
as  senior  management  considers  the  IRD  phase  to  be  an  activity 
leading  to  an  intangible  product.  Therefore  sufficient  time 
and  resources  are  not  allocated  to  the  IRD.  Budgetary 
constraints  and  the  desire  to  have  products  marketed 
frequently  lead  to  decisions  to  start  implementation  and  work 
out  the  details  or  problems  later.  However,  neglecting  the 
IRD  phase  can  prove  to  be  disastrous.  The  literature  is  full 
of  examples  of  systems  development  projects  that  were 
terminated  or  experienced  significant  delays  and  cost  overruns 
when  inadequate  resources  were  placed  into  the  IRD  phase.  The 
following  are  just  a  few  examples: 


•  Army  Civilian  Personnel  System  experienced  a  cost  increase 
from  the  original  65  million  projected  to  96  million 
dollars  (GAO,  1989) . 

•  Defense  Logistics  Agency's  Defense  Logistic  Service  Center 
experienced  a  cost  increase  from  the  original  123  million 
projected  to  177  million  dollars  (GAO,  1989) . 

•  Air  Force  Contract  Data  Management  System  experienced  a 
cost  increase  from  the  original  34  million  projected  to  74 
million  dollars  (GAO,  1989) . 

•  Navy's  Standard  Automated  Financial  System  experienced  a 
cost  increase  from  the  original  33  million  projected  to 
479  million  dollars  before  the  project  was  terminated 
(GAO,  19  89)  . 


Therefore,  the  IRD  phase  should  be  considered  to  be 
crucial  for  successful  systems  development.  The 
responsibility  of  convincing  management  to  provide  sufficient 


time  and  resources  for  the  IRD  phase  lies  with  the  systems 
analyst.  Furthermore,  it  is  also  the  systems  analyst's 
responsibility  to  ensure  that  developmental  efforts  during  the 
IRD  phase  are  competently  and  professionally  conducted. 

Each  subject  was  given  the  same  task  to  perform.  The  task 
involved  a  case  study  (Appendix  B)  of  a  utility  company's 
customer  order  processing  system.  This  case  study  had  been 
successfully  used  several  times  before  in  settings  involving 
the  use  of  protocol  analysis  for  identifying  problem  solving 
behavior  (Ramesh,  1989). 

The  task  was  to  design  a  customer  order  processing  system 
that  utilized  a  centralized  telephone  answering  service 
center,  connected  by  an  online  computer  to  a  large  number  of 
field  stations  that  were  responsible  for  providing  services  to 
the  customers  in  their  respective  geographical  areas  of 
responsibility.  The  objective  of  the  system  was  to  provide 
quick  response  to  customer  requests  for  activities  such  as 
starting  a  service,  stopping  a  service,  switching  a  service, 
repairing  appliances,  setting  up  a  location  for  services, 
removing  services  from  a  location  and  responding  to  power 
outages.  The  system  should  also  provide  real-time  status  of 
customer  requests.  The  status  would  be  used  by  utility 
representatives  at  the  centralized  telephone  answering  service 
center  for  answering  customer  inquiries  about  customer 
requests . 


Included  with  the  task  was  a  description  of  the  utility- 
company's  customer  order  processing  system  that  was  developed 
based  on  information  obtained  by  a  large  systems  consulting 
firm  during  an  actual  systems  development  project (Ramesh, 
1989)  . 

E.   EXPERIMENTAL  PROCEDURES 

The  experimental  sessions  were  scheduled  to  last 
approximately  three  hours.  A  separate  session  was  scheduled 
for  each  subject.  A  facilitator  was  present  for  each  entire 
session.  The  subjects  could,  if  desired,  take  a  short  recess 
during  the  session. 

Each  subject  was  provided  with  a  copy  of  this  information. 
The  subjects  were  advised  that  the  facilitator  possessed  more 
detailed  information  about  the  required  system  than  that  which 
was  provided  in  their  handout  and  could  answer  questions  for 
clarification  purposes.  This  option  was  only  available  prior 
to  the  initiation  of  the  design  process. 

1.   Subject  Briefing 

The  first  action  during  each  session  was  to  brief  the 
subject  about  what  he  would  be  doing  during  the  session  and 
how  the  data  he  was  going  to  provide  would  be  used.  The 
subject  was  then  assured  that  the  objective  of  the  research 
was  to  understand  rather  than  evaluate  the  problem  solving 
behavior. 
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2.  Think-Aloud  Familiarization 

The  subject  was  then  asked  to  participate  in  a 
structured  exercise  to  provide  him  with  the  opportunity  to 
become  comfortable  thinking -aloud  while  performing  a  task. 
First  he  was  given  instructions  designed  to  have  him  talk- 
aloud  while  performing  a  task.  Next,  he  was  provided  with  a 
task  (math  and  word  problem)  to  perform.  If  additional 
practice  was  needed  after  the  first  task,  subsequent  tasks 
(word  problems)  were  provided  until  the  subject  was 
comfortable  with  the  technique. 

3 .  Problem  Introduction 

The  subject  was  then  given  a  handout  with  a 
description  of  the  system  to  be  designed.  He  was  informed 
that  the  facilitator  had  additional  information  and  could 
clarify  issues  upon  request.  Then  he  was  asked  to  read  the 
handout  and  hold  any  questions  until  finished  with  the  initial 
reading.  At  that  point  any  questions  would  be  answered.  Any 
clarification  desired  would  be  provided.  Once  the  question 
and  answer  session  was  over,  the  subject  was  informed  that  he 
could  retain  the  handout  during  the  actual  design  process,  but 
that  the  facilitator  would  no  longer  answer  questions.  He  was 
also  advised  that  the  facilitator  would  be  present  only  to 
remind  him  to  think- aloud.  It  was  suggested  that  if  he  had 
additional  questions  he  should  make  whatever  assumptions  were 
necessary  to  proceed  with  his  assigned  task. 
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4.  Designing  the  System 

Designing  the  system  was  the  data  gathering  segment  of 
the  research.  it  was  at  this  point  that  the  tape  recorder 
was  turned  on. 

The  facilitator  was  present  during  the  entire  two-hour 
session.  However,  as  stated  before,  his  only  function  was  to 
remind  the  subjects  to  think-aloud. 

5.  Session  Wrap-Up 

The  subject  was  asked  to  fill  out  the  personal 
information  sheet  located  in  Appendix  C  at  the  end  of  the 
session.  The  information  requested  pertained  to  the  subject's 
educational  and  professional  background.  It  also  included  a 
question  about  whether  they  had  ever  participated  in  a  study 
that  required  them  to  think-aloud  while  performing  a  task. 

F.   TRANSCRIPTION  PROCESS 

All  transcribing  was  performed  by  the  facilitator.  Each 
subject's  entire  recorded  protocol  was  transcribed  verbatim. 
Punctuation  of  the  protocols  was  based  on  the  facilitator's 
knowledge  of  word  combinations  and  interpretation  of  the 
subjects'  verbal  inflections  while  speaking. 
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G.   ENCODING  PROCESS 

The  results  of  the  transcription  were  then  used  to  develop 
a  process  model .  Once  the  process  model  has  been  developed  it 
can  be  used  for  determining  the  subjects  problem  solving 
behavior. 

To  develop  a  process  model  from  which  to  analyze  the 
subjects  problem  solving  behavior,  the  weakest  hypothesis 
required  is  that  the  subjects'  cognitive  process  during  task 
performance  is  similar  to  the  operation  of  an  information 
processing  system  (Ericsson  and  Simon,  1984) .  Data  stored  in 
the  memory  of  a  computer  is  in  the  form  of  electrical  charges 
representing  either  ones  or  zeros.  Without  coding  the  ones 
and  zeros  are  meaningless,  but  when  encoded  they  represent 
data  that  is  recognizable  to  a  user.  The  human  brain  also 
represents  stored  data  as  electrical  charges.  For 
verbalization  or  output  to  take  place  the  data  in  memory  must 
first  be  encoded.  By  segmenting  the  verbalizations  into  their 
basic  components  and  then  coding  them,  one  can  use  the  coded 
data  to  develop  a  process  model  to  be  used  for  analysis. 

Protocols   are   frequently   analyzed   using   episode 

representation.   This  involves  splitting  the  protocols  into 

topic  segments  representing  a  single  task,  and  then  analyzing 

these  segments  to  identify  specific  kinds  of  activities  or 

episodes  (Ramesh,  1989) . 

Episodic  memory  contains  information  about  individual 
experiences  that  a  person  has  had  plus  generalized 
episodes  or  types  of  events.  Generalized  episodes  ...  are 
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created  by  comparing  different  episodes  and  calculating 
similarities  between  them. . .  According  to  Kolodner 
(19  83)  ,  "even  if  a  novice  and  an  expert  have  the  same 
semantic  knowledge  ...the  expert's  experience  would  have 
allowed  him  to  build  up  better  episodic  definitions... 
(Gilhooly,  pp.  184-186,1986) 


In  this  research,  episode  representation  was  used  with  a 
minor  variation  to  analyze  the  protocols.  Rather  than 
splitting  the  protocols  into  topic  segments  representing  a 
single  task  and  then  identifying  the  specific  episodes  within 
the  task  segments,  the  episodes  were  identified  and 
categorized  directly  from  the  protocols  without  any 
preprocessing . 

After  the  segmentation  was  completed  each  episode  was 
analyzed  independent  of  all  other  episodes  and  assigned  a 
number  representing  the  type  of  activity  captured.  This 
process  was  conducted  twice.  Results  were  compared  and 
differences  were  reviewed  before  final  assignment. 

The  numbering  system  for  the  types  of  episodic  activity  is 
provided  in  Table  1. 

After  all  episodes  were  categorized,  the  numeric  episode 
representations  were  separated  from  the  protocols  for 
comparison  between  the  subjects. 
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Table  1:  EPISODE  TYPES  KEY  CODE 


EPISODE  TYPE 

KEY  CODE 

Identification  of  Relevant  Data 

1 

Identification  of  Problems 

2 

Identification  of  Relationships  between 
Problems 

3 

Generation  of  Hypothesis 

4 

Confirmation  of  Hypothesis 

5 

Assessment  of  Problem  Criticality 

6 

Goal  Setting 

7 

Reviewing  Previous  Actions 

8 

Requesting  Clarification  from  Customer 

9 

Making  Assumptions 

10 

Disregarding  Previous  Actions 

11 

Reading  from  Case  Write-Up 

12 

Intuitive  Leap 

13 
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III.   RESEARCH  FINDINGS  AND  DISCUSSION 

A.   INTRODUCTION 

During  the  encoding  process,  the  following  pattern  of 
activities  were  observed.  The  subjects  frequently  performed 
independent  activities,  within  the  same  episodic 
classification,  for  an  extended  period.  The  numeric 
representations  of  these  episodes  were  first  examined  for 
consistency  in  coding  within  the  activity  period.  This  was 
done  primarily  for  verification  of  the  coding  validity.  The 
results  of  the  evaluation  of  the  coding  scheme  within  the 
different  activity  periods  were  found  by  the  research 
facilitator  to  be  consistent  throughout  the  entire  process. 

The  next  activity  was  to  examine  episodic  activity  in 
sequence  from  first  activity  to  last  activity.  The  results  of 
this  indicated  that  all  subjects  performed  the  same  types  of 
activities  in  essentially  the  same  order  throughout  the  design 
process.  A  global  evaluation  of  the  numeric  representation 
confirmed  this  conclusion,  and  indicated  that  the  problem 
solving  activity  was  an  iterative  process.  The  pattern  of 
episodic  activity  indicated  that  the  subjects  had  divided  the 
design  process  into  different  levels  and  worked  at  each  level 
in  the  same  manner. 
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Based  on  the  data  encoded  the  remaining  parts  of  this 

chapter  focuses  on  the  development  of  the  process  model  and 

the   specific   behaviors   of   the  subjects   during   task 
performance . 

B.   PROCESS  MODEL  DEVELOPMENT 

The  design  process  used  by  novice  systems  analysts 
consists  of  four  phases.  These  four  phases  are  the  goal 
formulation  phase,  identification  of  relevant  data  phase, 
hypothesis  generation  phase  and  hypothesis  confirmation  phase. 
A  graphic  model  of  this  process  is  provided  in  Figure  1. 

1.   Goal  Formulation 

The  initial  activity  engaged  in  by  each  of  the 
subjects  was  goal  formulation.  These  goals  were  broad 
statements  of  intended  actions.  The  goals  were  mostly 
concerned  with  the  procedural  aspects  of  developing  a  data 
flow  diagram  instead  of  identification  of  problems  and 
developing  solutions. 

While  goal  formulation  was  the  initial  activity  of 
each  subject,  it  was  not  restricted  to  the  initial  period  of 
activity.  Development  of  the  initial  goals  into  sub-goals 
occurred  throughout  the  design  process.  The  sub-goals  were 
also  primarily  concerned  with  procedural  aspects  or  the 
technicalities  of  the  data  flow  diagram  development  process. 
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Figure  1:  Novice  Systems  Analysts'  Process  Model 

However,  at  the  sub -goal  level  there  were  some  goals  that  were 
concerned  with  the  performance  of  activities  within  an  element 
of  the  data  flow  diagram  such  as  a  specific  process  like 
"start  a  service".  This  dynamic  process  of  formulating  goals 
and  sub-goals  was  probably  the  result  of  the  iterative 
processes  associated  with  the  development  of  a  data  flow 
diagram. 

It  was  stated  earlier  that  the  goals  of  the  subjects 
were  very  broad.  This  vagueness  coupled  with  the  magnitude  of 
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the  task  and  the  numerous  details  that  had  to  be  accounted  for 
may  have  contributed  to  the  difficulty  that  all  subjects 
experienced  in  keeping  track  of  events  not  being  directly 
acted  on  at  the  time. 

2.   Identification  of  Relevant  Data 

In  this  phase,  the  subjects  extracted  and  examined  in 
extensive  detail  all  the  information  contained  in  the  customer 
interview  write-up.  The  relevant  information  was  categorized 
as  entities  (internal  or  external) ,  data,  or  processes  in  the 
system.  The  subjects  then  set  out  to  establish  relationships 
between  the  data,  the  entities  and  the  system  in  a  format 
which  would  accommodate  the  desired  functionality. 

It  was  during  this  phase  that  problems  were 
encountered.  These  problems  were  associated  with  how  to 
represent  data  or  how  a  specific  event  would  affect  a  larger 
whole.  As  previously  stated  they  were  very  specific  problems 
of  limited  scope.  These  problems  were  either  resolved  when 
discovered  or  set  aside  for  future  use. 

The  subjects'  interest  in  data  was  primarily 
quantitative.  The  search  for  data  was  a  broad-based  search. 
The  attitude  was  look  at  everything,  because  all  data  were 
perceived  as  important.  So  much  effort  was  expended  looking 
at  all  of  the  data  that  there  was  little  energy  left  for 
looking  at  any  data  in  detail.  Data  were  for  the  most  part 
taken  at  face  value.   For  example,  There  was  an  entity  called 
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"customer".  However,  the  customer  could  be  a  new  customer  or 
an  existing  customer.  The  utility's  accounting  and 
engineering  departments  could  also  be  customers.  None  of  the 
subjects  was  able  to  designed  their  system  to  include  all  the 
customer  types  through  all  the  processes.  Usually,  only  two 
of  the  four  customer  types  were  considered  throughout  the 
entire  design  process.  However,  each  of  the  four  customer 
types  did  require  unique  processing  in  at  least  one  of  the 
requests.  This  meant  that  duplication  of  efforts  was  not  a 
valid  reason  for  overlooking  the  different  types  of  customers. 
3 .   Hypothesis  Formulation 

During  this  phase  the  subjects  developed  a  single 
hypothesis  for  the  specific  process  upon  which  they  were 
working  at  the  time.  The  hypothesis  developed  were  directed 
towards  getting  a  process  requested  by  the  customer  through 
the  various  levels  of  the  data  flow  diagram.  Hypothesis  were 
generated  for  processes  at  every  level  required  to  fully 
illustrate  the  processes  satisfactory  from  start  to 
completion. 

The  subjects  did  not  try  to  find  a  best  way  to  build 
a  system.  Their  action  was  to  find  any  way  to  build  a  system. 
They  would  generate  a  hypothesis,  and  if  that  hypothesis 
passed  testing,  it  was  accepted. 

Another  problem  was  that  the  subjects  would  generate 
a  hypothesis,  and  then  apparently  forget  about  it.  This  could 
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be  attributed  to  undefined  goals  or  it  may  have  been 
indicative  of  under  developed  indexing  abilities  in  the 
systems  analyst's  domain. 

4.   Hypothesis  Confirmation 

The  subjects  design  rational  concerning  hypothesis 
formulation  and  confirmation  was  very  direct  and  very  narrow 
in  scope.  The  first  action  was  to  formulate  a  hypothesis. 
The  subjects  then  determined  if  the  hypothesis  could  be  tested 
with  data  already  extracted  from  the  handout.  If  the 
hypothesis  could  not  be  tested  with  the  available  data,  the 
subjects  would  search  for  more  data.  If  the  hypothesis  could 
be  tested  with  the  available  data,  the  subjects  would  test  the 
hypothesis.  If  the  hypothesis  passed  testing,  the  subjects 
confirmed  the  hypothesis.  If  the  hypothesis  failed  testing, 
the  subjects  discarded  the  hypothesis  and  started  the  process 
over. 

The  testing  consisted  of  actually  running  the  process 
or  data  through  the  entire  developed  portion  of  data  flow 
diagram  and  verifying  that  all  the  necessary  events  occurred 
(desk- top  testing) .  If  the  hypothesis  passed  the  test,  it  was 
confirmed  and  the  subject  continued  with  the  iterative  design 
process.  If  the  hypothesis  did  not  pass  the  test  it  was 
discarded  and  the  subject  would  either  formulate  a  new 
hypothesis  right  then  or  return  to  the  data  gathering  phase. 
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C.   IDENTIFICATION  OF  THE  NOVICE  ANALYSTS'  BEHAVIORS 

Constructing  the  process  model  is  just  the  first  step  in 
the  analysis  of  the  problem  solving  behavior  of  the  novice 
systems  analyst  during  the  design  process.  In  many  ways  it 
can  be  considered  a  quantitative  evaluation,  providing  a 
global  perspective  from  which  to  view  the  problem  solving 
behavior.  But,  more  importantly  it  provides  an  organized 
framework  from  which  further  analysis  and  qualitative 
evaluation  of  specific  behaviors  can  be  conducted. 

The  results  of  this  qualitative  evaluation  provides 
meaningful  insights  into  the  problem  solving  behavior  of  the 
systems  analyst  during  the  design  process.  This  information 
may  be  compared  to  similar  information  about  novices  in  other 
fields  to  identify  systems  analyst's  unique  attributes 
(Ramesh,  1989) .  However,  the  greatest  benefit  from  the 
information  obtained  from  this  analysis  is  that  may  be 
compared  to  results  of  similar  studies  of  expert  systems 
analyst  to  identify  differences  in  expert -novice  problem 
solving  behavior  (Vitalari,  1981) . 

The  following  are  findings  from  the  qualitative  analysis 
of  the  process  model.  Novices: 

•  Set  broad,  poorly  structured  goals. 

•  Relied  on  their  ability  to  apply  the  data  to  the 
methodology. 

•  Frequently  referenced  the  handout  to  ensure  all  the  data 
was  accounted  for. 
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•  Used  a  iterative  design  methodology. 

•  Allowed  methodology  to  lead  them  through  the  design 
process  in  an  unorganized  fashion. 

•  Retained  the  same  methodology  throughout  the  design 
process . 

•  Recognized  the  importance  of  user  participation  in  the 
design  process. 

•  Recognized  their  shortcomings  and  the  need  to  consult 
others  about  their  activities. 

•  Did  not  use  heuristics. 

•  Did  not  have  preconceived  expectations.  They  asked  very 
few  questions  during  the  question  answer  period. 

•  Did  not  readily  recognize  when  errors  had  been  made. 

•  Spent  little  time  checking  their  work. 

•  Did  not  consider  intra-organization  politics  during  the 
design  process. 

•  Were  not  concerned  about  training  and  implementation. 

•  Accepted  data  at  face  value. 

•  Did  not  restrict  their  problem  space  to  a  manageable  size. 

•  Exhibited  a  lack  of  confidence  in  their  action. 

•  Did  not  evaluate  the  quality  of  their  hypotheses. 

•  Were  able  to  identify  complex  relationships  between  data. 

•  Were  unable  to  incorporate  their  initial  impressions  of 
firm  into  their  design  process. 

•  Used  desk- top  testing  to  confirm  hypotheses. 

•  Discarded  hypotheses  when  data  did  not  support  the  desk- 
top testing. 

•  Did  not  seek  problems  or  clues.  Novices  solved  problems 
as  they  were  encounter  or  set  them  aside  for  later 
resolution. 
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In  Chapter  IV  these  findings  are  compared  with  finding 
from  earlier  studies  to  identify  expert-novice  and  novice- 
novice  differences. 
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IV.   ARE  NOVICE  SYSTEMS  ANALYSTS  DIFFERENT  FROM  THEIR 
EXPERIENCED  COUNTERPARTS?   -  A  COMPARATIVE  STUDY 


To  understand  expert-novice  differences,  it  is  useful  to 
study  the  characteristics  of  expert  behavior  as  established  by- 
prior  work.  These  characteristics  can  then  be  compared  to 
those  of  novices  in  our  study  to  identify  areas  of  potential 
differences . 

A.   CHARACTERISTICS  OF  EXPERTS 

Expertise  is  most  often  exhibited  by  individuals  in  a 
single  domain  and  most  characteristics  that  experts  possess 
are  necessarily  domain  specific.  However,  the  following 
characteristics  have  been  noted  in  most  experts  regardless  of 
their  domain. 


•  Experts  are  specialists  and  exhibit  their  expertise 
primarily  in  the  domain  of  their  specialty.  Intelligent 
people  may  be  that  way  because  of  the  indexing  of  Long 
Term  Memory  in  their  domain  specialty  rather  than  the 
global  quality  of  their  thinking  (Minsky  and  Papert, 
1974)  . 

•  Experts  exhibit  superior  perceptual  skills  within  their 
domain.  As  shown  in  a  study  of  GO  players,  experts  are 
able  to  see  larger  patterns  than  novices  (Reitman,  1976) . 

•  Experts  are  faster  and  more  accurate  than  novices.  That 
can  be  the  shown  through  the  development  of  automatic 
skills  by  practice  such  as  in  typing  which  frees  up  memory 
capacity  for  the  other  task  essential  to  typing  such  as 
reading  or  taking  dictation  (Chi,  Glasser  and  Farr,  1988) . 
It  may  also  be  shown  in  the  large  meaningful  patterns  that 
experts   perceive   within   their   domain,   therefore, 
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eliminating  many  of  the  search  routines  that  novices  would 
have  to  perform.  Cab  drivers  will  recognize  a  shorter 
route  while  traveling  to  their  destination,  even  though 
the  route  was  not  identified  in  advance  in  a  laboratory- 
setting  (Chase,  1983)  . 

•  Experts  have  expanded  Short  Term  Memory  within  their 
domain  through  the  automaticity  of  many  portions  of  their 
skills.  Normal  Short  Term  Memory  is  10-18  digits, 
however,  trained  memory  experts  can  remember  up  to  80 
digits  in  short-term  serial  recall.  (Chase  and  Ericsson, 
1982)  . 

•  Experts  see  through  the  superficial  attributes  of  a 
problem  to  its  deeper  meaning.  Expert  programmers  will 
group  programs  by  algorithms  while  novices  will  group  them 
by  their  application  (Weiser  and  Shertz,  1983). 

•  Experts  will  spend  a  lot  of  time  before  starting  to  solve 
the  problem,  just  thinking  about  it,  in  order  to  obtain  a 
better  understanding  of  how  they  should  go  about  solving 
the  problem  .  Novices  just  tend  to  plunge  in  immediately 
and  work  towards  a  solution  haphazardly  (Chi,  Glasser  and 
Farr,  1988)  . 

•  Experts  will  check  their  work  more  often  than  novices  and 
seem  to  better  recognize  when  they  have  made  errors  that 
need  correction  before  continuing  (Chi,  Glasser,  and  Farr, 

1988) . 


Ramesh  (1989)  used  episodic  knowledge  obtained  from  the 
transcriptions  of  the  verbal  protocol  to  develop  process 
models  for  representing  the  problem  solving  behavior  of  expert 
and  novice  financial  analysts.  He  found  that  expert  financial 
analysts  solved  their  problems  in  three  distinct  phases 
referred  to  as  the  situation  assessment  phase,  hypothesis 
generation  phase,  and  the  diagnostic  evaluation  phase.  Within 
these  distinct  phases  specific  activities  were  exhibited  by 
the  expert  financial  analysts.  The  situation  assessment  phase 
included  goal  formulation,  examination  of  key  areas,  and 
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formulation  of  initial  impressions.  The  hypothesis  generation 
included  select  and  analyze  relevant  data.  Finally,  the 
diagnostic  evaluation  phase  consisted  of  identification  of 
causes,  criticality  assessment  and  identification  of  other  key 
areas . 

These  findings  will  be  compared  and  contrasted  with  the 
results  of  the  analysis  from  this  study  to  identify  any 
expert -novice  differences. 

Vitalari  (1981)  used  operator  representation  technique, 
involving  the  identification  and  use  of  knowledge  elements  and 
operator  elements  to  study  systems  analysts'  problem  solving 
behavior.  He  categorizes  expert  behavior  into  different  types 
of  search  behavior,  problem  perspective  and  focus. 

The  different  types  of  search  behavior  are  the  search  for 
triggers  or  clues,  search  for  goals  which  limit  the  magnitude 
of  the  search  area,  search  for  different  strategies  from  which 
to  approach  the  problem  in  order  to  maintain  flexibility,  and 
the  search  for  applicable  hueristics  used  for  reducing  the 
search  time  (Vatalari,  1981). 

Experts   were   determined   to   have   a   global   problem 

perspective  (Vatalari,  1981) .    This  was  determined  to  be 

consistent  with  the  concepts  of  hierarchical  decomposition  and 

modularity  associated  with  structured  analysis  techniques 

(Gane  and  Sarson,  1979) . 

The  focus  was  primarily  concerned  with  the  different 
orientations  from  which  to  evaluated  and  solve  the  problem. 
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There  was  the  report -orientation,  job-orientation,  political- 
orientation,  involvement -orientation,  management -orientation 
and  facilitation  focus  (Vatalari,  1981) . 

Expert -novice  differences  will  be  identified  by  comparing 
Vitalari's  (1981)  findings  concerning  how  expert  systems 
analysts  developed  and  executed  their  search  behaviors, 
problem  perspectives  and  basic  orientations  to  approaching  the 
problem  to  findings  in  this  study  of  novice  systems  analysts' 
problem  solving  behavior. 

B.   CHARACTERISTICS  OF  NOVICE  SYSTEMS  ANALYSTS  AND  DESIGN 

Ramesh  (1989)  found  that  novices,  like  experts,  solved 
problems  in  three  distinct  phases.  These  phases  were 
different  from  that  which  the  experts  used  in  their  problem 
solving  behavior.  The  phases  were  problem  identification, 
hypotheses  formulation  and  final  diagnosis.  Each  phase 
included  different  activities.  The  problem  identification 
phase  included  goal  identification  and  indication  of  problems. 
The  hypothesis  formulation  phase  activities  were  group  related 
findings  and  identify  consistent  findings.  The  final 
diagnosis  phase  activities  consisted  of  identification  of 
causal  linkages  and  final  diagnosis  (Ramesh,  1989). 

The  problem  solving  behavior  of  novice  financial  analysts 
identified  in  Ramesh' s  (1989)  study  was  compared  to  problem 
solving  behavior  of  novice  system  analyst  identified  in  this 
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study    to    identify    novice-novice    differences    and 
characteristics  unique  to  novice  system  analysts. 

The  novice-novice  differences  that  were  identified  are 
presented  in  the  following  section. 

C.   NOVICE -NOVICE  DIFFERENCES 

Novice-novice  differences  refers  to  the  different  behavior 
exhibited  by  members  from  different  domains  while  performing 
a  domain  specific  task.  In  order  to  identify  novice-novice 
differences,  novices'  protocols  from  different  domains  are 
analyzed  using  the  same  method  of  representation.  This 
methodology  enables  the  researcher  to  identify  the  differences 
in  the  problem  solving  behavior  between  the  two  domains  and 
also  allows  for  generation  of  hypothesis  about  behavior  which 
may  be  unique  to  a  specific  domain  or  common  to  all  domains. 

While  analysis  of  the  problem  solving  behavior  of  members 
from  two  different  domains  is  too  limited  a  field  to  confirm 
hypotheses  about  the  uniqueness  of  domain  specific  problem 
solving  behavior,  the  analysis  of  the  problem  solving  behavior 
of  members  from  two  different  domains  does  allow  for  the 
formulation  of  hypotheses  that  can  be  investigated  in  future 
studies . 

1.   Systems  Analysts -Financial  Analysts  Differences 

Financial  analyst  problem  solving  behavior  consisted 
of  the  problem  identification,  hypothesis  formation  and  final 
diagnosis  phases  (Ramesh,  1989)  .    This  differed  from  the 
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problem  solving  behavior  of  novice  systems  analysts  in  which 
four  distinct  phases  consisting  of  goal  formulation,  gathering 
of  relevant  data,  hypothesis  formulation  and  hypothesis 
confirmation  were  identified. 

Identifying  symptoms  or  clues  to  problems  to  be  solved 
was  a  primary  consideration  of  financial  analysts  (Ramesh, 
1989)  .  This  activity  was  not  important  to  the  systems 
analysts.  The  systems  analysts  were  interested  in  obtaining 
data  to  support  the  development  of  a  design  following  a 
methodology.  Problems  that  systems  analysts  encountered  were 
a  hinderance  to  task  performance,  and  not  the  primary 
consideration  of  the  systems  analysts.  They  were  something 
that  needed  solving  so  that  designing  the  system  could 
continue . 

Goal  formulation  and  identification  of  problems  are 
episodic  events  within  the  financial  analysts'  problem 
identification  phase  (Ramesh,  1989).  The  financial  analysts' 
goals  were  not  well  defined  and  used  generic  terms  (Ramesh, 
1989).  The  novice  systems  analysts'  goals  were  not  any 
clearer  or  better  defined  goals,  however,  they  were  technical 
in  content.  Novice  systems  analysts'  goals  indicated  that 
they  were  going  to  commence  an  activity  that  was  specific  and 
structured.  While  novice  systems  analysts  did  not 
specifically  state  what  they  were  going  to  do  within  framework 
of  that  structured  methodology,  the  fact  that  they  were  going 
to  participate  in  a  very  structured  activity  implies  that  the 
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primary  orientation  of  novice  systems  analysts  is  technical 
with  secondary  consideration  given  to  managerial  factors  such 
as  the  business  environment  and  the  overall  strategy  of  the 
organization.  This  is  different  from  the  novice  financial 
analysts  who  are  concerned  primarily  with  managerial  issues 
such  as  health  of  the  firm,  business  environment  and  strategy. 
The  novice  financial  analysts  use  ratios,  graphs  and 
statistical  analysis  as  tools  to  aid  managers  in  their  job  of 
achieving  the  company  goals. 

The  primary  difference  in  the  hypothesis  formulation 
phase  was  that  the  financial  analysts  would  formulate  a 
hypothesis  which  had  to  be  justified  by  significant  findings 
(Ramesh,  1989)  .  If  the  findings  did  not  lead  to  the 
formulation  of  a  plausible  hypothesis,  then  the  data  would  be 
discarded.  The  novice  systems  analysts'  hypotheses  did  not 
require  significant  findings  in  the  Hypothesis  Formulation 
phase  to  justify  its  validity.  In  the  Hypothesis  Formulation 
phase  of  the  novice  systems  analysts  nothing  was  discarded. 

The  hypothesis  confirmation  phase  of  novice  systems 
analysts  differs  from  the  final  diagnosis  phase  of  novice 
financial  analysts  in  several  ways.  The  primary  activity 
within  the  hypothesis  confirmation  phase  of  novice  systems 
analysts  is  the  testing  of  the  formulated  hypothesis.  It  is 
during  this  phase  that  a  hypothesis  will  be  discarded  if  data 
do  not  support  it.  There  is  no  similar  activity  within  the 
Final  Diagnosis  phase  exhibited  by  novice  financial  analysts. 
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Also  systems   analysts  do  not   attempt   to   integrate  the 
findings.   The  systems  analysts'  task  is  an  iterative  process 
within  the  framework  of  the  formal  structured  methodology. 
The  fully  developed  structure  is  the  final  product. 
2.   Systems  Analysts  Specific  Events 

Systems  Analysts  were  unique  in  at  least  two  aspects. 
During  goal  formulation  the  novice  systems  analysts  committed 
to  a  formal  structured  methodology  for  performing  the  assigned 
task.  Second,  after  the  systems  analysts  formulated  goals 
they  would  gather  data  and  formulate  one  hypothesis  that  fit 
the  data  to  the  structure  and  then  test  the  hypothesis.  A  the 
successful  testing  resulted  in  hypothesis  confirmation.  This 
format  in  which  the  novice  system  analysts  would  formulate  a 
hypothesis,  test  the  hypothesis  and  then  confirm  or  discard 
the  hypothesis  based  on  the  ability  of  the  data  to  support  the 
hypothesis  was  unique  to  systems  analysts. 

D.   EXPERT -NOVICE  DIFFERENCES 

The  next  sections  describe  the  expert-novice  differences 
obtained  from  comparisons  of  novice  systems  analysts'  problem 
solving  behaviors  with  expert  financial/systems  analysts' 
problem  solving  behaviors. 

1.   Episode  Representation  Differences 

The  comparison  of  novice  systems  analysts  and  novice 
financial  analysts  discussed  earlier  indicated  that  there  was 
sufficient  similarity  for  comparison  between  the  two  groups. 
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Ramesh's  (1989)  work  with  expert  financial  analysts  using 
episode  representation  to  model  and  analyze  their  problem 
solving  protocols  provides  an  excellent  opportunity  for  such 
a  comparison.  In  this  section  we  will  compare  behavior  of 
expert  financial  analysts  and  novice  systems  analysts.  The 
major  differences  between  the  two  categories  are  as  follows: 


•  Experts  stated  their  goal  clearly  and  referred  to  them 
frequently.  Novice  goals  were  broadly  stated,  technical 
in  wording  and  not  frequently  referred  to  after  initially 
stated. 

•  Experts  started  their  analysis  with  an  initial  impression 
of  the  firm  which  helped  them  form  expectations  and 
develop  list  of  potential  problem  areas.  Novices  form 
initial  impressions  of  the  firm  and  then  disregarded  them 
once  the  design  process  started. 


Experts  Spend  a  Great  Deal  of  Time  Analyzing  a  Problem 
Qualitatively.  Protocols  show  that,  at  the  beginning  of 
a  problem- solving  episode,  experts  typically  try  to 
"understand"  a  problem,  whereas  novices  plunge  immediately 
into  attempting  to  apply  equations  and  to  solve  for  an 
unknown.  What  do  the  experts  do  when  they  qualitatively 
analyze  a  problem?  Basically  they  build  a  mental 
representation  from  which  they  can  infer  relations  that 
can  define  the  situation,  and  they  add  constraints  to  the 
problem....  (Chi,  Glasser  and  Farr,  pp.  xvii-xx,  1989) 


Experts  searched  for  clues  to  support  or  negate  their 
hypothesis  during  the  hypothesis  generation  phase. 
Novices  accepted  their  hypotheses  at  face  value.  Novice 
system  analysts  did  not  evaluated  their  hypotheses  until 
the  Test  Hypothesis  phase. 

Experts  looked  for  "good"  and  "bad"  signals  in  the  data 
they  analyzed  to  be  used  for  correcting  initial  hypothesis 
in  case  there  were  discrepancies.  When  a  novice 
formulated  a  hypothesis  it  was  accepted  as  is  or 
discarded.  The  quality  of  the  hypothesis  was  not  a 
consideration . 
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•  Experts  were  able  to  identify  and  maintain  in  their  minds 
complex  relationships  between  various  segments  while 
analyzing  data.  Novices  could  identify  complex 
relationships  within  segments  and  across  various  segment, 
but  could  not  maintain  the  relationships  while  analyzing 
the  data. 

•  Experts  had  preconceived  ideas  of  "key  areas"  that 
required  examination.  This  was  probably  the  result  of 
past  experience  in  solving  similar  problems.  Novices  did 
not  assign  special  meanings  to  particular  areas,  but 
rather  looked  at  all  areas  in  the  order  that  was  presented 
in  the  case  write-up.  Novices  have  very  limited  practical 
experience,  therefore,  they  are  not  able  to  generated 
preconceived  ideas  of  what  to  expect  or  look  for,  but  must 
look  at  every  thing  in  total . 

•  Experts  ranked  their  hypothesis  in  order  of  importance. 
Novices  did  not  rank  their  hypothesis.  There  were 
indications  that  the  novices  recognized  that  certain 
hypothesis  were  more  important  than  others,  but  these 
hypothesis  were  not  given  any  special  consideration. 

•  Experts  were  interested  in  the  firm's  goals  and 
strategies,  both  internal  and  external  to  the 
organization.  Novices'  interests  in  the  firm  were  related 
to  how  the  customer  service  system  would  interact  with 
different  functional  areas  internal  to  the  organization. 

•  Experts  were  interested  the  managerial  style  of  the  firm 
and  the  business  environment  in  which  the  firm  operated. 
Novices  were  only  interested  in  the  system  which  they  were 
designing. 


2.   Knowledge  Operator  Differences 

Nicholas  P.  Vitalari  (1981)  used  knowledge  operator 
representation  to  analyze  the  problem  solving  protocols  of 
expert  and  novice  system  analysis.  The  following  behavioral 
differences  were  the  results  of  a  comparison  between  the 
findings  of  this  study  and  Vitalari 's  work.   Experts: 


Set  specific  well  structured  goals.  Novice  set  broad 
poorly  structured  goals. 
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•  Developed  multiple  strategies  for  achieving  the  goals. 
Novices  did  not  develop  a  strategy.  They  allowed  the 
methodology  to  lead  them  through  the  design  process  in  an 
unorganized  fashion. 

•  Would  modify  or  discard  a  strategy  when  evidence  indicated 
that  the  goals  were  not  being  achieved.  Novices  stayed 
with  the  same  methodology  throughout  the  process. 

•  Applied  heuristics  during  the  problem  solving  process. 
There  was  no  indication  that  the  novices  used  heuristics 
at  any  time  during  the  process. 

•  Were  aware  and  showed  consideration  for  the  political 
attributes  of  the  firm  such  as  territorial  consideration 
and  orders  of  priority  on  who  initiated  and  responded  to 
various  actions.  Novices  recognized  that  there  were 
different  areas  of  authority,  but  treated  each  as  an  equal 
partner. 

•  Were  concerned  about  the  organizational  behavior  such  as 
the  "resistance  to  change"  phoneme.  Novices  were  only 
interest  in  designing  a  system.  They  did  not  consider  how 
this  would  affect  the  organization. 

•  Were  concerned  about  the  skills  of  the  individuals  within 
the  organization.  Novices  did  not  show  any  interest  in  the 
skills  of  the  people  within  the  organization.  There  was 
no  concern  for  staffing  requirements  for  operating  and 
maintaining  the  system  after  implementation. 

•  Were  interested  in  how  the  secondary  data  could  be 
formatted  and  used  by  upper  management  in  addition  to  the 
uses  at  the  operational  level.  Novices  were  only 
concerned  with  getting  a  workable  system  to  the 
operational  level. 

•  Showed  concern  for  more  than  the  end  product.  They  were 
interested  in  things  like  peoples  names  showing  a  personal 
orientation.  Novices  were  only  interested  in  the  product. 

•  Searched  for  clues  about  the  problem  for  use  in 
formulating  goals  and  strategies.  The  novices  took  all 
the  data  at  face  value. 

•  Had  expectations  and  their  search  is  characterized  by  what 
is  missing  instead  of  what  is  apparent.  Novices  search 
for  data  is  a  rehashing  of  what  is  apparent. 

•  Were  interested  in  the  organization  structure  for  use  in 
developing  maps  and  strategies  to  use  during  the  problem 
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solving.  Novices  relied  on  the  structure  of  the 
methodology  and  how  raw  data  fit  into  the  structure. 

Used  goals  to  manage  the  size  of  the  problem  space. 
Novices  did  not  set  good  goals  and  it  was  apparent  from 
the  data  loss  during  the  design  process  that  the  problem 
space  was  larger  than  the  novice  could  effectively  manage. 

Exhibited  behavior  which  is  meant  to  reduce  uncertainty. 
Novices  exhibited  a  lack  of  confidence  in  their  ability  to 
design  the  system.  Their  action  increased  the 
uncertainty. 
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V.   CONCLUSION 

This  study  of  the  problem  solving  behavior  of  novice 
system  analyst  had  three  objectives: 

•  To  analyze  the  problem  solving  behavior  of  the  novice 
system  analysts  during  the  design  process. 

•  To  compare  the  results  of  this  study  to  those  of  previous 
studies  involving  experts  and  identify  the  differences. 

•  To  suggest  possible  uses  for  the  differences. 

Protocol  Analysis  provided  a  methodology  for  the  analysis 
of  the  problem  solving  behavior  of  novice  system  analysts. 
Similar  studies  have  been  conducted  using  experts.  Ramesh's 
(1989)  studies  of  expert  financial  analysts  using  episode 
representation  and  Vitalari's  (1981)  work  with  expert  systems 
analysts  using  knowledge  operator  representation  were  two  of 
these  studies. 

The  findings  of  this  study  suggest  that  there  are 
significant  differences  between  expert  and  novices,  in  the  way 
they  perform  systems  analysis  and  design.  Suggested  ways  to 
use  these  differences  are  provided  in  the  following 
paragraphs . 

The  first  and  most  obvious  use  for  the  findings  is  in  the 
traditional  instructor- student  relationship.  By  calling  the 
experts  behavior  the  goal  and  the  novices  behavior  the  current 
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situation,  educational  programs  can  be  designed  and 
implemented  to  practice  those  specific  things  that  experts  do 
and  that  novices  do  not  do.  Specifically  efforts  are  directed 
towards  known  activities  in  which  experts  participate. 

The  second  and  more  remote  use  of  the  findings  is  in  the 
development  of  intelligent  tutoring  systems.  This  method  of 
education  does  not  use  a  human  instructor,  but  rather  the 
student  and  a  machine  interact  in  a  learning  environment. 

Currently,  the  bottleneck  in  the  development  of  effective 
expert  systems  is  in  the  area  of  adequately  modeling  the 
behavior  of  experts  for  the  systems.  The  availability  of  the 
behavioral  models  and  expert -novice  differences  could  assist 
with  the  development  of  expert  or  intelligent  tutoring  systems 
for  systems  analysts. 

Third,  CASE  technology  currently  provides  users  with  tools 
that  automatically  generate  code,  create  and  maintain  a  data 
dictionary,  write  and  update  all  required  documentation  and 
performs  other  tasks  currently  associated  with  systems 
development  just  by  using  various  data  flow  and  entity 
diagrams  as  input.  The  output  is  consistent  and  technically 
correct,  however,  the  output  is  only  as  good  as  the  diagrams 
that  are  used  as  input  to  the  CASE  tool.  If  the  quality  of 
the  analysis  and  diagram  development  used  for  input  to  the 
CASE  tool  are  poor,  then  the  resultant  output  is  a  consistent 
and  technically  correct  poor  system. 
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A  potential  use  for  the  results  of  this  research  is  to 
build  an  expert  system  as  a  front -end  processor  to  a  CASE 
tool.  The  front -end  processor  would  be  used  to  evaluate  the 
level  of  expertise  of  the  analyst  using  the  tool.  By 
understanding  the  differences  between  the  skills  of  the 
analyst  providing  the  input  and  an  expert  analyst's  skills, 
the  front -end  processor  can  improve  the  quality  of  input 
thereby  ensuring  the  code  and  other  generators  operate  using 
the  best  possible  input. 
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Appendix  A:  AN  OVERVIEW  OF  PROTOCOL  ANALYSIS 

A.  PROTOCOL  ANALYSIS  DEFINED 

A  generic  definition  of  Protocol  Analysis  is  given  below. 

Protocol  analysis  is  a  general  term  for  the  collection  and 
analysis  of  verbal  reports  made  by  subjects  while  they 
perform  a  task.   (Vitalari,  p.  81,  1981) 

Ericsson  and  Simon  (1984)  broaden  the  scope  of  this 

definition  by  allowing  for  two  types  of  verbalization.   The 

first  is  a  verbal  report  where  subjects  talk  aloud  while 

performing  a  task,  referred  to  as  concurrent  verbalization. 

The  second  type  of  verbalization  is  a  report  made  by  subjects 

subsequent  to  the  performance  of  a  task,  referred  to  as 

retrospective  verbalization. 

B.  OVERVIEW  OF  THE  COGNITIVE  PROCESS 

A  primary  assumption  behind  Protocol  Analysis  is  that  an 
individual's  problem  solving  behavior  can  be  determined  from 
analysis  of  the  subject's  verbalizations.  These 
verbalizations  consist  of  interactions  between  information 
contained  in  Short  Term  Memory  (STM)  and  Long  Term  Memory 
(LTM)  . 

1.   Short  Term  Memory 

STM  is  a  recording  of  events  and  stimuli  which  an 
individual  has  experienced  recently.  STM  capacity  is  minimal, 
though  there  is  some  variance  between  individuals.   It  is 
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instantly  available,  not  requiring  retrieval  delay.  STM 
duration  can  vary  from  that  of  a  fleeting  nature  to  moments. 
However,  the  longer  the  information  that  an  individual  is 
consciously  aware  of,  or  "heeded"  remains  in  STM  the  greater 
the  probability  that  it  will  become  contaminated  by  the 
information  previously  stored  in  LTM,  thereby  losing  its 
original  identity.  This  does  not  preclude  the  possibility 
that  the  "heeded"  data  may  have  already  been  stored  in  LTM 
prior  to  contamination.  The  probability  of  data  storage  in 
LTM  is  dependent  upon  the  duration  that  it  is  "heeded"  in  STM. 


It  is  even  more  obvious  that  information  can  be  retrieved 
from  LTM  only  if  has  been  stored  there  previously,  and 
retained.  The  hypothesis  about  storage  (fixation)  that 
seems  to  us  most  defensible  in  the  light  of  the  empirical 
evidence  is  that  if  and  only  if  information  is  heeded  in 
STM  for  a  sufficient  interval  of  time  will  it  be  stored 
(and  indexed)  in  LTM.  (Ericsson,  Simon  and  Herbert,  p.  81, 
1984) 


Finally,  STM  is  the  gateway  through  which  all  interactions  - 
passive  and  active  with  LTM  are  routed. 
2  .   Long  Term  Memory 

LTM  is  an  accumulation  of  all  the  events  or  stimuli 
that  an  individual  has  experienced  and  "heeded"  during  their 
life.  Access  to  LTM  compared  to  STM  is  slow  yet,  its  capacity 
is  virtually  unlimited. 

Demonstrated  ability  to  retrieve  obscure  facts  from 
the  voluminous  amount  of  information  that  an  individual  has 
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stored  in  LTM  over  a  lifetime  of  experiences  seems  to  imply- 
that  LTM  is  indexed  in  some  fashion.  This  indexing  or 
categorization  in  many  instances  is  highly  organized  and  can 
be  rapidly  retrieved  as  demonstrated  in  studies  of  the  "tip  of 
the  tongue"  (James,  189  0)  and  "felling -of -knowing"  (Woodworth, 
193  8)  phenomena. 

A  simplified  view  of  accessing  or  retrieving 
information  from  this  indexed  or  categorized  LTM  is  to 
visualize  a  set  of  symbols  formulated  in  STM  which  cause 
electrical  impulses  to  activate  locations  in  LTM.  As  a  result 
data  in  the  activated  locations  are  copied  into  STM.  This 
process,  depending  on  a  variety  of  factors,  can  be  almost 
instantaneous,  approaching  the  time  required  to  access  STM  or 
it  can  proceed  over  an  extended  period.  The  degree  of 
accuracy  of  the  data  retrieved  from  LTM  can  vary.  This 
variance  occurs  because  the  indexing  or  categorization  is 
dependent  upon  the  relationships  between  data  stored  in  LTM. 
The  longer  the  data  are  stored  in  LTM,  the  greater  the 
probability  that  the  boundaries  between  the  relationships  that 
characterize  different  but  similar  data  will  merge.  This 
merging  of  boundaries  results  in  data  stored  in  LTM  acquiring 
characteristics  not  present  when  initially  "heeded" . 
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C.   PROTOCOL  ANALYSIS  AS  A  RESEARCH  METHOD 

Protocol  Analysis  is  a  valid  methodology  for  identifying 
the  cognitive  processes  of  individuals  during  task  performance 
(Ericsson  and  Simon,  1984)  .  It  provides  the  researchers  with 
several  ways  to  analyze  problem  solving  behavior. 

1.   Types  of  Protocol  Analysis 

Protocol  analysis  is  typically  conducted  using  two 
different  techniques.  The  techniques  can  be  used  separately, 
or  in  conjunction  with,  each  other.  The  first  method, 
retrospective  analysis,  requires  the  subject  to  perform  a 
task,  and  after  the  task  has  been  completed  the  subject 
verbalizes  what  he  is  thinking  about  as  he  performs  the  task. 
The  second  method,  concurrent  analysis,  requires  the  subject 
to  verbalize  as  he  is  performing  the  task.  When  the  two 
methods  are  used  in  conjunction,  the  subject  is  asked  to 
verbalize  while  performing  the  task,  and  then  after  task 
completion  the  subject  is  asked  to  reflect  on  his  thoughts 
during  task  performance. 

The  effectiveness  of  these  two  analysis  techniques  are 
briefly  discussed  below. 

a.   .Retrospective  Analysis 

Retrospective  analysis  is  very  susceptible  to 
inaccuracies.  The  primary  reason  is  that  retrospective 
analysis  requires  the  subject  to  verbalize  about  an  event  that 
has  occurred  sometime  in  the  past.   If  the  event  that  the 
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subject  is  responding  to  occurred  in  the  recent  past,  portions 
or  even  all  of  the  "heeded"  data  may  still  reside  in  STM. 
However,  as  stated  earlier,  the  possibility  for  contamination 
of  data  in  STM  increases  as  the  duration  of  "heeded"  data  in 
STM  increases.  If  the  data  to  be  verbalized  are  no  longer  in 
STM,  then  the  subject  has  to  initiate  the  retrieval  of  the 
desired  information  from  LTM  to  STM  for  encoding  and 
verbalization.  In  summary,  information  retrieved  from  LTM  may 
be  inaccurate  for  several  reasons: 


The  information  that  has  been  stored  in  LTM  is  not  always 
initially  retrievable  in  total. 

Over  time,  the  boundaries  in  LTM  weaken  resulting  in  the 
merging  of  data  of  a  similar  nature. 

The  subject  may  misunderstand  what  is  requested  and 
provide  inaccurate  data,  even  though  it  may  be  relevant. 


b.      Concurrent  Analysis 

Concurrent  analysis  is  not  as  susceptible  to  the 
inaccuracies  associated  with  retrospective  analysis.  Since, 
the  subject  verbalizes  his  thought  process  as  he  performs  the 
task,  there  is  little  chance  that  the  information  is 
contaminated. 

2.   Issues  in  Using  Protocol  Analysis  as  a  Research  Method 

The  purpose  of  this  thesis  is  to  use  protocol  analysis 

to  analyze  the  problem  solving  behaviors  of  novice  systems 

analysts  during  task  performance.   However,  there  are  several 
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concerns  about  the  validity  of  protocol  analysis  as  a  research 
method  that  need  to  be  addressed  first.   These  concerns  are: 


•  Doubts  have  been  expressed  about  using  verbalizations  as 
data  (Ericsson  and  Simon,  1984) . 

•  Concerns  have  been  raised  about  the  spontaneity  of  the 
subjects  verbalization  (Ericsson  and  Simon, 1984). 

•  It  has  been  suggested  that  the  verbalizations  are  "soft" 
data  e.g.  they  are  subjective  and  not  measurable  (Ericsson 
and  Simon,  1984) . 

•  Theoretical  presuppositions  during  the  encoding  process 
have  to  be  considered  (Ericsson  and  Simon,  1984) . 

•  Can  the  subject  be  trusted  to  not  to  be  deliberately 
misleading  with  their  verbalizations  (Ericsson  and  Simon, 
1984) ? 


The  following  sections  address  these  concerns. 

a.  Response   to  Doubts 

The  doubts  expressed  by  many  psychologists  about 
the  suitability  of  subjects  verbalizations  as  scientific  data 
have  to  do  with  introspection.  Psychologists  argue  that 
retrospective  analysis  is  a  variant  of  the  discredited  process 
of  introspection  (Ericsson  and  Simon,  1984).  However,  there 
is  a  significant  difference  between  protocol  analysis  and 
introspection.  Protocol  analysis  requires  the  subjects  to 
verbalize  their  thoughts  as  they  occur.  This  is  possible 
because  of  the  invention  and  availability  of  tape  and  video 
recorders.  On  the  other  hand,  introspection  involves  a 
trained  domain  specialist,  usually  a  psychologist,  reflecting 
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on  his  own  problem  solving  activities  and  writing  or  dictating 
to  a  stenographer  an  analysis  of  these  activities.  This 
process  has  all  the  previously  mentioned  problems  associated 
with  retrospective  analysis,  in  addition  to  any  preconceived 
biases  of  the  person  doing  the  introspection. 
Jb.  Influencing  Verbalization 

To  obtain  data  suitable  for  the  type  of  protocol 
analysis  selected  for  use,  a  methodology  must  be  developed  to 
influence  the  subject's  verbalization.  Through  careful 
wording  of  the  instructions,  different  kinds  of  verbalizations 
can  be  obtained  from  a  subject.  Examples  of  the  instructions 
are: 


"I  would  like  you  to  say  out  loud  what  ever  thoughts  come 
into  your  mind,  as  if  you  were  in  a  room  alone  talking  to 
yourself .  " 

"I  would  like  you  to  talk  aloud  as  you  are  performing  the 
task. " 

"I  would  like  you  to  verbalize  what  you  are  doing  as  you 
are  doing  it  and  why  you  are  doing  it" 


The  first  two  constructions  are  structured  to  obtain  responses 
suitable  for  concurrent  analysis.  The  third  construction 
seeks  to  obtain  a  hybrid  response  for  use  in  concurrent  and 
retrospective  analysis. 

c.  "Soft"  Data  Versus   "Hard"  Data 

Data  collected  by  verbalization  are  considered 
"soft"  data  primarily  by  people  who  misunderstand  how  the 
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verbalizations  are  going  to  be  interpreted.  "Soft"  data  are 
obtained  when  data  interpretation  occurs  simultaneously  with 
the  verbalizations.  By  using  this  method  the  interpretation 
is  subjected  to  the  theoretical  presuppositions  of  the 
interpreter  (Ericsson  and  Simon,  1984) . 

"Hard"  data  can  be  collected  and  analyzed 
objectively.  An  example  of  "hard  data"  would  be  the  recording 
of  the  time  between  eyelid  movements  during  task  performance. 
When  using  protocol  analysis  the  protocols  are  recorded  and 
transcribed  verbatim.  This  data  are  just  as  "hard"  as  the 
data  about  eyelid  movement  (Ericsson  and  Simon,  1984) . 
d.      Theoretical   Presuppositions  During  Encoding 

With  protocol  analysis,  a  subject  speaks  out  loud. 
The  verbalization  is  recorded  and  then  transcribed  verbatim. 
At  this  stage  there  are  no  interpretations  and  no  inferences, 
with  the  exception  of  some  processing  as  verbal  inflections 
and  emphasis  are  represented  by  punctuation. 

To  interpret  "hard"  data  obtained  from  subjects, 
verbalizations  need  to  be  encoded.  The  coding  process,  just 
by  its  very  nature,  is  subject  to  some  theoretical 
presuppositions.  The  objective  when  analyzing  the  subjects 
verbalizations  is  to  minimize  the  theoretical  presuppositions 
to  facilitate  providing  an  objective  as  possible  analysis  of 
the  subjects'  behavior.  This  is  done,  first  by  minimizing 
inferences  and  expectations  used  by  the  person  developing  the 
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coding  scheme.  Further,  it  is  accomplished  by  ensuring  the 
person  or  persons  doing  the  actual  coding  of  the  protocols 
were  not  involved  with  the  development  of  the  coding  scheme. 
Finally,  the  protocols  should  be  coded  by  at  least  two  people 
working  independently.  The  results  of  the  codings  are  then 
compared  and  any  differences  are  resolved. 
e.      Trusting  the  Subject 

A  basic  but  absolutely  essential  concern  is  how  to 
ensure  that  the  subjects  are  not  purposely  misleading 
researchers  during  the  process  (Ericsson  and  Simon, 1984) .  If 
the  subject  was  asked  in  an  introspective  report  about  his 
mental  state  or  thought  processes,  trust  is  important  and  some 
validation  is  crucial  (Ericsson  and  Simon,  1984)  .  With 
protocol  analysis,  researchers  can  eliminate  the  need  to  trust 
the  subjects  and  collect  the  data  required  for  determining 
problem  solving  behavior  during  task  performance  during  the 
same  process. 


However,  the  issue  of  the  reliability  of  self- 
reports  can  (and,  we  think,  should)  be  avoided 
entirely.  The  report  "x"  need  not  be  used  to  infer 
that  X  is  true,  but  only  that  the  subject  was  able  to 
say  "X"_(i.e.,  had  the  information  that  enabled  him  to 
say  "X" . )  by  following  this  path,  we  can  even  show 
that  there  is  an  inverse  relation  between  how  much 
subjects  need  to  be  trusted  and  how  much  information 
they  verbalize.  For  the  more  information  conveyed  in 
their  responses,  the  more  difficult  it  becomes  to 
construct  a  model  that  will  produce  precisely  those 
responses  adventitiously- hence  the  more  confidence  we 
can  place  in  a  model  that  does  predict  them. 
(Ericsson,  Simon  and  Herbert,  p. 7,  1984) 
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D.   SUMMARY 

Verbalization  is  an  activity  of  the  cognitive  process  that 
is  recordable  for  analysis.  Protocol  analysis  provides  the 
means  to  capture  and  analyze  verbalizations  during  task 
performance  in  order  to  identify  problem  solving  behavior.  It 
is  a  methodology  in  which  minimal  theoretical  presupposition 
is  required  for  use,  thereby,  increasing  the  objectivity  of 
the  analysis  of  the  data.  When  using  protocol  analysis  the 
issue  of  trusting  the  subjects  is  not  a  concern.  These 
factors  make  protocol  analysis  a  valuable  tool  that  should 
continue  to  be  used  and  refined. 
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Appendix  B:  CUSTOMER  SERVICE  SYSTEM 

Southeast  States  Power,  a  gas  and  electric  utility  company,  is 
automating  a  major  part  of  its  customer  service  system.  The 
proposed  system  should  utilize  a  centralized  telephone 
answering  service  center,  connected  by  an  online  computer  to 
a  large  number  of  field  stations.  The  central  system  will  be 
installed  with  the  objectives  of  providing  quick  processing  of 
requests  and  providing  accurate  information  to  customers  about 
the  status  of  their  requests.  In  the  centralized  set  up, 
calls  from  customers  will  be  routed  to  any  clerk  available  at 
the  service  center.  Further,  requests  for  service  may 
originate  from  within  the  utility  itself.  The  accounting 
department  may  request  cancellation  or  restoration  of  services 
based  on  the  payment  profile  of  customers.  The  engineering 
department  may  request  setting  up  or  removal  of  services  from 
a  location. 

The  proposed  system  will  support  the  following  types  of 
request : 

•  starting  a  service 

•  stopping  a  service 

•  switching  a  service  from  one  location  to  another 

•  setting  up  a  location  for  service 

•  removing  service  from  a  location 
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•  restoring  service  to  a  customer  with  local  outage 

•  restoring  service  to  customers  after  system-wide  outage 

•  maintaining  and  repairing  appliances 

The  clerks  generate  service  orders  based  on  the  request 
for  service.  Records  maintained  by  the  system  contain 
information  about  the  customers,  location,  equipment  and 
appliances  at  the  location, a  and  status  of  system- wide 
service.  The  clerks  retrieve  relevant  information  from 
internal  records  while  generating  service  orders.  The  clerks 
submit  the  service  orders  to  the  service  center  supervisors 
for  their  authorization.  the  supervisors  forward  authorized 
service  orders  to  field  station  supervisors  for  follow  up. 

While  processing  a  request  from  a  customer  for  starting  a 
service,  the  clerk  obtains  authorization  from  the  credit 
department.  Further,  the  clerk  ascertains  the  need  for 
requesting  a  deposit  from  the  customer.  If  the  customer  is 
requesting  that  the  service  be  stopped,  the  clerk  should 
terminate  any  contracts  for  maintenance  of  appliances  that 
have  been  entered  into  with  the  customer.  The  requests  for 
switching  service  from  one  location  to  another  will  involve 
starting  a  service  in  the  new  location  and  stopping  the 
service  at  the  old  location.  While  processing  the  request  for 
setting  up  and  removing  service  from  a  location,  the  clerk 
obtains  details  about  the  location  and  equipment  used  in  the 
location  so  that  internal  records  can  be  updated.   When  a 
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customer  reports  an  outage,  the  status  of  the  system- wide 
service  should  be  checked  before  orders  are  placed  for  follow 
up.  When  requests  for  repairs  of  appliances  are  received,  the 
clerk  checks  appliance  maintenance  contracts  to  ascertain 
charges  (if  any)  for  customer  billing. 

Field  station  supervisors  set  priorities  on  service  orders 
and  then  group  them  based  on  the  location  and  nature  of 
service  to  be  provided.  Field  technicians  with  necessary 
skills  (such  as  appliance  repair,  equipment  set  up  etc.)  are 
dispatched  to  provide  service.  After  providing  the  service, 
field  technicians  report  the  status  of  the  order  (such  as  the 
nature  of  the  service  provided  and  completion  status)  to  the 
field  supervisors.  Field  supervisors  reconcile  this 
information  with  the  service  orders.  Appropriate  charges  (if 
any)  are  computed  and  communicated  to  the  accounting 
department  for  customer  billing. 

The  clerks  provide  information  about  the  status  of  the 
service  requests  upon  inquires  from  customers.  They  also 
notify  service  center  supervisors  when  complaints  about 
delayed  or  inadequate  services  are  received. 

CUSTOMER  SERVICE  SYSTEM  PROCESS  DESCRIPTION 

CALCULATION  OF  DEPOSITS 

Compute  the  deposit  to  be  paid  by  the  customer  based  on 
the  type  of  customer  (residential  or  industrial) ,   the  past 
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consumption  pattern  (if  available,  is  maintained  for  a  12 
month  period)  or  expected  consumption  as  follows: 

Industrial  Customer 

The  deposit  required  is  based  on  the  maximum  monthly- 
consumption  from  the  consumption  history.  The  actual  value 
is : 

max_consumption  *  a  constant  (say  1.5)  *  approximate  rate  per 
unit  (say  $0.1  per  unit). 

When  the  history  is  not  available,  the  maximum  monthly 
consumption  is  computed  as  proportional  to  the  installed 
capacity  of  the  industrial  unit.   The  actual  value  is: 
Installed  capacity  *  a  constant  (say  2.0)  *  approximate  rate 
per  unit  (say  $0.1  per  unit) . 

Residential  Customers 

the  deposit  required  is  based  on  the  average  monthly 
consumption  from  the  consumption  history.  The  actual  value 
is  : 

average  consumption  *  a  constant  (say  2.0)  *  approximated  rate 
per  unit  (say  $0.08  per  unit) 

When  the  history  is  not  available,  the  average  monthly 
consumption   is   based   on   the   type   and   size   of   the 
house/apartment  to  which  the  service  is  provided.   Expected 
consumption  is  maintained  as  a  table  of  the  following  form: 
Type  of  House   Size  (room)    Consumption 
Town  House       4  300  units 

Condo  2  100  units 
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The  actual  value  is: 
Average  consumption  *  a  constant  (say, 2.0)  *  approximate  rate 
per  unit  (say,  $0.8  per  unit) 

(The  following  information  was  not  given  to  the  subject.  It 
was  used  for  clarification  during  the  question  answer 
session. ) 

CUSTOMER  SERVICE  SYSTEM  CLARIFICATION  FROM  CLIENT  INTERVIEW 

The  proposed  system  will  support  the  following  types  of 
request : 


•  starting  a  service:  starting  a  gas/electric  service  as  a 
customer's  location. 

•  stopping  a  service:  stopping  a  gas/electric  service  at  a 
customer's  location  when  the  customer  moves  out  of  the 
area  serviced  by  the  utility  company. 

•  switching  service  from  one  location  to  another:  when  a 
customer  moves  from  on  location  (old  location)  to  another 
location  (new  location)  that  is  also  service  by  the 
utility  company,  service  is  discontinued  in  the  old 
location  and  started  in  the  new  location. 

•  setting  up  a  location  for  service:  setting  up  a  location 

(  such  as  installing  appliances,  meters  etc.)  so  that 

gas/electric  service  can  be  provided  to  customers  who  will 

move  into  the  locations.   The  requests  for  setting  up  a 

location  originate  from  the  engineering  department. 

•  removing  service  from  a  location:  removing  equipment  such 
as  appliances  or  meters  from  a  location  so  that  the 
location  will  not  be  serviced  by  the  utility.  The 
requests  for  removing  service  from  a  location  originate 
from  the  engineering  department. 

•  restoring  service  to  a  customer  with  local  outage:  local 
outages  are  outages  that  affect  the  customer  only. 
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•  restoring  service  to  customers  after  system-wide  outage: 
system-wide  outages  are  outages  that  affect  a  large  number 
of  customers. 

•  maintaining  and  repairing  appliances:  stoves,  furnaces, 
ovens  etc.  are  examples  of  appliances. 

•  Each  field  station  provides  service  to  part  of  the  total 
geographical  area  serviced  by  the  utility. 

•  Field  supervisors  obtain  information  about  the  status  of 
the  orders  from  technicians  and  enter  this  information  in 
the  system.  This  status  information  is  available  to 
telephone  answering  center  clerks  for  answering  queries 
from  customers. 


TELEPHONE  SERVICE  CENTER 


The  clerk  receives  requests  from  customers  as  well  as  the 
accounting  and  engineering  departments  of  the  utility. 
Activities  leading  to  the  generation  of  requests  at  the 
engineering  and  accounting  departments  is  beyond  the  scope 
of  the  system. 

The  clerks  processes  the  requests  and  sends  them  to 
supervisors  for  authorization  and  further  action. 

The  clerks  have  access  to  credit  information  about  the 
customers  (that  is  created  an  maintained  by  the  credit 
department) .  This  credit  information  specifies  whether 
any  deposit  is  required  from  the  customer  while  processing 
the  request.  The  amounts  of  deposit  is  calculated  by  the 
system  using  a  pre- specif ied  formula.  The  clerk  informs 
the  customers  the  amount  of  deposit  and  payment  plans. 
The  billing  is  handled  by  the  accounting  department. 

When  new  customers  whose  credit  information  is  not 
available  request  that  a  service  be  started,  the  clerk 
send  a  request  for  credit  authorization  to  the  credit 
department.  Further,  the  clerk  processes  the  request  from 
the  customer  and  forwards  it  to  the  supervisor  for  follow 
up. 

A  customer  will  have  separate  billing  accounts  for  each  of 
the  locations  in  which  he/she  gets  service.  The  clerk 
obtains  the  name  and  address  of  the  customer  or  the 
account  number  to  process  requests. 
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A  customer  may  call  to  ascertain  the  status  of  a  request 
made  earlier.  if  the  customers  complain  about  delays  in 
service  or  inadequate  services,  the  clerk  forward  these 
complaints  to  the  supervisors. 

Status  of  system-wide  service  is  available  to  the  clerks. 
Customer  complaints  information  is  not  used  in  determining 
whether  a  system-wide  problem  exists. 

For  each  appliance  installed  at  a  customer's  location, 
information  on  their  type,  installation  date,  service 
history,  maintenance  contract  details  etc.  is  maintained. 
The  clerk  checks  the  maintenance  contract  information  to 
determine  whether  customer  needs  to  be  billed  for 
appliance  repairs. 

Telephone  answering  center  supervisors  authorize  the 
service  order  and  forward  them  to  field  supervisors. 

The  credit  department  forwards  credit  authorization  for 
new  customers  to  the  supervisors.  Supervisors  contact  the 
customers  when  authorization  information  is  received  to 
inform  them  the  status  of  their  request  as  well  as  deposit 
requirements . 


FIELD  STATION 


Field  station  supervisors  assign  priorities  to  service 
orders  according  to  a  pre -determined  scheme. 

Supervisors  assign  one  technician  to  process  each  order. 

When  the  technician  report  on  the  status  of  the  order, 
their  supervisors  reconcile  the  information  with  the 
service  orders.  further,  based  on  the  information,  the 
system  computes  appropriate  charges  (if  any)  according  to 
a  pre -determined  formula  for  customer  billing. 

The  information  on  the  status  of  the  service  orders  is 
available  to  telephone  answer  center  clerks  for  answering 
queries  from  customers. 
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Appendix  C:  PERSONAL  DATA  QUESTIONNAIRE 

1.  What  is  your  name? 

2 .  What  is  your  branch  of  service? 

3.  What   is   your   community?   (Aviation,   Surface-Line, 
Submarines,  Supply,  Intelligence,  other)? 

4.  What  is  your  curriculum  and  current  quarter? 

5.  What  was  you  undergraduate  major? 

6.  Have  you  had  and  computer  training  prior  to  attending 
Naval  Post  Graduate  School?  If  yes,  was  it  design, 
programming,  databases,  other?  How  many  quarters  or 
semesters? 

7.  Have  you  ever  worked  on  Systems  Development  Project?  If 
yes,  in  what  capacity? 

8.  Have  you  ever  been  a  subject  in  a  study  utilizing  protocol 
analysis? 
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